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RESEARCH AND EVALUATION IN EDUCATIONAL TELEVISION 



Categories; The Impe 
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but Indlspens'able Tools 
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A story that has 6erved long and* well to illustrate a basic 
point in semantics goe^ something like this: Three baseball umpires 
were discussing how they reach decisions when judging baseball games. 
The first umpire said, "Some are strikes"; some are balL^ and I call 
them as they are." The secd'nd umpire said', "Some are^ striken J some 
are balls; and I call them as I s^e them." The third umpire said, ^ 
"Some may be called strikes and some may be called balls, but they 
aren't anything uritil I call them." 

In talking abput research/evaluation instead of baseball, the 
"different views of teality repre&er^tfed by fe;he umi^ires in the story are 
still with us. The first umpire would; probably say that some project ^ 
is or is not research, d,s if he wete malcing a simple factual statemen£. 
The second umpire holds th^ same basic beliefs as the first, excfpt he • 
recognlTzes that his observat^^onal -^powers are not infallible. It is oiily 
umpire three who appreciates fi^lly the creative power of the label or 
category. Given-.the variety of i;'esearch labels now in use throughout 

. ' . ■ ■ :\ . . 

the literature, the research categ\^ries used in this chapter should be 
viewed through the eyes of umpire number thr^e; not as having the qual- 
ity of correctness , but, hopefully, as having the quality of utility. 

Overview of the Chapter 

First, a fundamental distinction is ma<ie^etween basic re- 
search (seeking new, generalizablg knowledge) and adminir«"rative research 
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providing information useful to decision-makers who hav.a a mission to 
perform). Attention is then given to four types o'£ administrative 
research: background research, formative research, smranative research, 
and policy research. Having thus subdivided a complex process into 
"analytic categories, a concluding "series of questions and comments, 

about a hypothetical ETV program will put the separate parts back into 

< f . 

their natural inter- relationships . 

The disproportionate emphasis on administrative research does 
reflect a value ^judgmenl: , ntft. on the superiority of one research method 
or philosophy over another, but on what will be most relevant to the 
needs of an ETV administrator. Administrative research should help • 
people make decisions and solve problems. The various ' categories o^ 
Research discussed here, therefore, should serve as an inventory bf ways 
that re-search can be useful. ' 



Basic Research 



In general, ETV organizations are not directly involved in 
basic communication research; therefore the bulk of this chapter will 



not deal directly with this categoj^r. Basic research is, however, a 
- --Qseful point of reference for describing,, contrasting, and understand-' 
ing the administratively oriented research more typical of ETV .opera- 
tions, so it will be discussed first, 

Basic research is -sometimes (Tailed "pur^^'* research or 
theoreticarresearch. 4 distingui sluing characteristic is its primary 
purpose: increasing generalizable knowledge and understanding of basic 
principles through the donstruction, testing, and validation of theories 
(explanations' of relationships).^ Predictions are derived" logically from 



tlie theories for empirical tests in a researcli\design; Vjxe major 
research' criteria are internal validity (Can the results be Interpreted 
unambiguously?), external validity (Are the findings generallzable?) , 
and theoretic relevance (Was the right question asked irk the first 
pllce? Was the theoretic explanation actually tested Ifi this set of 



observations?) • 



be in getting the 
represented so that 



In basic research, stimulus materials would ^e valued, not 

\ . ' I 

for what they are, but for the theoretical conditions |that they 
represent . Suppose a basic researcher is studying tejtevised violence 
and effec'fcs. As part of his experlmen^tal requlre|tiients , he uses a 
film clip of a brutal fight scene. Does this researcher have any 
interest in ^hls particular fight scene per se? No;! it Jjas chosen from 
any number of alternative film clips that could represent the needed 
degre^ of violence. The researcher's Interest would 
required levels of vix)lence clearly and efficiently 
his experimental results can be unambiguous. The pajrticular fight scene 
employed, and/or the movie from which it was-taken^ is of little or no 

ultimate interdsK to the basic researcher. Later, it will be se6n that 

i - 

this contrasts markedly with the administrative researcher, whose 

interest will be in a particular program as a program, not as ^^replace- 

i • . 

aile representative of some theoretic condition. 

Basic research thrives in a disciplinary ^vironment, where it 

/ O 

serves to develop th^ discipline. Minimal condjftibns needed to estab- 
lish a discipline would appear to be: / (1)C^ cohesive set of variables 
with which the discipline\^ill be con^t^^ed; (2) an accepted research 
methodology or set of methodologies Ased to explorfe relationships among 



those variables; ^an\j (3) a body o/^lheoretical knowledg^^^tKat describes 
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and explains in generalizable terms the relationshi 

„„ , relationships among varia.bles 

^s..u..,., ^^^^^^^^ ^^^^ 

--puna. B../.,„e.V..e„.e..: „.e,,c. 

^ M.„, aucpunar, „„ae.p.„„.„,, -^^^^ 7 " 

psychology, social psycholoEv , 

etc., plus a host of creative anH r V 

" ' 

Of ...abliahad dlsclpuoas, b« not a .o.al „Ua„ k 

. ^ reliance on them for the 

basic research to devleoo tha ,.u . 

evleop the theories and principles needed in applied 
communication organisations such as EW M 

especlallv h " ^^^i^Principles and^practice 

J ^ P-ction poetic., .em to he of onl. pe.pHeral interest 

:z:r r^''''^' - - - 

8 fUld Of c™M„„i.atl„„ rasearch, .hi. ^^j, j,,,^ 

-xaelned a ^s. o( haalc raaearch ,„aa.l„„a .hat . 

,, ' lusstions that seek to probe sden- 

cauy the „at.e a„d pote.lal o, .dla a.h as tele.alo. .p.^ 

t r ri ^--^-^ °^ uteta. 

-a^^ "plloltly .th .edl/^„„ 
istlcs la the work Of Gavrui sawn (e g salo . 
Salomon. 1970, 19>2). ' = 

R»ther than engaging primarily m b.=( 

"research, EIV oreani 

nations typlcallV have some social ao« ■ 

S„„ » ° ="'Sned to them, 

-etlmes the progra^er can find gnidance for his day-to-day 

~ .™ ba-sic theory and research, sometimes not. „ , 

and if the programmer has been assigned the mission Of 
reducing illitericy then ^ho ^^, " 

y. then the theoretical insights of the basic 
researcher should be of great nr..^• , 

Sreat practical value to the ETV programmer. , 
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In other situations, it may be far more difficult to determine the 

direct implicationa for ETV po,licy and practice from basic research, 

* - . 

s, • 

Administrative Research 

Techniques of inquiry can be- and are used for purposes "^other 
than basic research, theory construction, and disciplinary development. 
Such techniques are also means of generating the data and information 
required for rational decision-making in administrative structures such 
as ETV organizations. This entire category of research designed to 
assist decision-making is being called ajiministrative research . 

There are some general characteristics that can distinguish . 
administrative research from the basic research discuss^ed earlier, iThe 
basic researcher sets his .own constraints (the limits of his problem, 
the techniques to be employed, the factors to study, the factors to 
ignore, etc), for the most part. The administrative researcher gene- 
rally starts his work with most of these constraints assigned to him. 
Decision-making requirements dictate the problem and set constraints on 
what can be an acceptable answer. 

The basic researcher deals with some form of the question: 

4 

{ 

What is the effect of X on Y? Contrast this with the ETV adrainistra- 
tive researcher's typical questions: 'Is there a need for this program? 
Is there an audience for. this program? Will the target audience pay 
attention to this program?: Will they comprehend it? vls^one replay 
enough? Should t.tye continue the program next year? Has the program 
met its objectives? 

Guttentag (1971) writes: "Evaluation research always involves 
a judgment of the worthwhi loness of some activity. At the outset, 
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therefore, it is quite different Trom'the explicit value-free position 
of experimental research" (p. 76). (Note. that Guttentag's statement * 
does not Imply a value-free position of the experimental researchers , 
themselves.) In some form, administrative decision-making, and the 
administrative research that backs itiup, deals with "the best way" (a 
value position) to allocate resources in order tomeet some objective 
(with the objective being an accepted value position). All components 
of the organization, research included, are manifestations of a general 
commitment to find the "best" way to do a "good" thing. 

Administrative research (sometimes called evaluation) deals 
with decision-making. There are also decisions to he made In basic 
research, of course, but they are formalized in advance to conform with, 
scientific procedures for hypothesis acceptance or rejection. Ideally,- 
In theory construction and: disciplinary development, basic researchers 
should have the ability, even the responsibility, to hold commitments 
and beliefs in abeyance until there is sufficient supportive experimental 
evidence. Typically, the administrative researcher will not have the 
option to hold decision-making in abeyance. There is usually a finite 
set of options, with4he decision-maker having to choose from that set 
"on the basis of all the evidence, logic, intuitions, expe?ience, advice, 
and value judgments at his command. 

In basic research, there is more or less a sole reli-ance on 
evidence that has met the disciplinary requirements of acceptability. 
In 'administrative decision-making, the research evidence is one input 
alongside several others to be considered in the decision-making process. 

In such "mother" disciplines as psychology, the controlled 
experiment is regarded highly, and right'fuiily so (see, e.g., Campbell, 
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1957; Ker linger, 1965). To reduce extraneous influences, the basic 
researcher will devise stringent controlsXthat acjte^t to hold all 
things equal or accounted for in two or more groups except some care- 
(y fully manipulated factor . That "factor" will be a/ theoretic variable 



that is expected to make a difference. If a difference does occur, the 
good de*^ign will allow one to attribute the diffe'rence to the manipulated 
independent variable, because alternative explanations of the difference 
will have been mrade unlikely^ through caxeful controls. Although an 
oversimplified summary statement of experimental logic, this does allow 
us to contrast the administrative research situation. Through the eyes 
of a basic researcher in psychology, the ETV administrative researcher 
would not have sufficfent control over the stimulus (the TV progr^uoal 

the subjects (the: audienceX or the students), the design (the conditions 

\\ ^* 

of exposure), or even the de^ndenC variable measurements (the criterion 
tests). The introduction' of ETV usually me^ns a whole new system gor 
doing things. Not oi^, but many new factors or variables are involved. 
These factors are confounded and interrelated, and they make it very 
difficult to apply experimental log^. To a basic researcher, require- 
ments of a rigorous design would ^erride in importance the basic mission 
of a TV program. To an administrative researcher in ETV, maintenance of 
c the basic mission of an ongoing program or series would override the 

demands of a 'rigorous experimental design. 

All research, basic or administrative, costs money, but the 
conscious consideration of cost vs. 'payoff in research s^ms to be 
greater in administrative research. Conceivably^ almost any reasonable 
administrati\/e datum can be acquired for a price. The value' judgment 
the administrator must make is how much of the systerft's resources can 

ERLC - 8 



be expended to find out something. This hits home immediately in ETV 
when, fbr example, a $10,000 program might cost half that much-*o, 
evaluat*^ properly. Hierein lies one reason (but no^the only reason) 
for a paucity of true experimental conditions being met in ETV adminis- 
trative research. Sometimes administrative research is criticized for 
methodological compromises as if the researchers were unaware of what 
more ideal conditions might be, when in fact the compromise was dictated 
by expediency and economics. There is a considerable difference between 
the researcher who compromises in ignorance and the researcher who knows 
y good research procedures and design, but must reluctantly deviate from ' 
the ideal in order to accomplish a mission. . 

For purposes of administrative research in ETV, classical 
experiments fail on several counts. Consider the following: 

1. In exerting experimental control over the stimulus (the 
TV program), it may be simplified to" the point where it no longer repre- 
sents real world situations. Experiments that utilize visuals consist- 
.ing of stick figures or simple line drawings, for example, are suspect 
in this regard. v 

/ 2. In an experimental design set up td measure effects of, 
say, an entire serifes of ETV programs, ot^^^ctors such as reception 
conditions, post-viewing reinforcement, etc., would have to be held 
constant throughout the series. If changed in midstream, it would be 
difficult to separate program effects from effects of the other sources 
•of .variance. Typically, an administrator uou^d not tolerate this holding 
pattern if he saw something obviously going wrong. He would^feJl duty 
bound to improve his program as soon as pos^sible a'nd as much as possible. 
The experiment, per se, could yield no interim feedback for. purposes of ' 
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corrective chattge. Tkl experiment, then, can yield an after-the-fact^ 
Judgment or "post mortem" on the series, but this is probably not the 
primary informatiorF' need of 'the administrator. . 

3. ^ In some experimental designs, the control group require- 
ment means .that a certain group should receive no treatment. This can 
bet an ethical ^dilemma for the administrator who^ almost by definition^ 
believes in the value of his program. If ^the program^s to bring good 
and needed results, should a group be deprived of exposure just to per- 
mit a clean-cut experimental assessment of effects? In ETV as well ^s 
in medicink and crime prevention, etc., the administr>t;or * s under- 
standable response would often have to be "no." 

4. Repeatedly, the requirements of tight design are impossi- 
ble to implement in ongoing programs. For instance, subjects should be 
drawn from a target population and assigned to experimental treatment 
groups at random . This' is .usually impossible in situations where/ the 
researcher must work at the^^nvenience of the education program, 
rather than vice versa. i 

5. The very rigor that gives experimental design its power 
also serves to shut o^t information that comes in farms outside the 
scope of the experiment. That is,* experimental conclusions will be on 
the^asis of quantitative measurement of the' dependent variable. Any, 

insights that are not reflected in that tieasurement l^ave no effect on 

<^ ' 

the conclusion. 

At least on the surface, then, there seems to be a great amount 
of incompatibility between the values of basic research and the values : 
of administrative research. Indeed, several have stated in some fashion 
that good basic research tends to make bad administrative research 
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(Gutteptag, 1971; Stuff lebeam' 1967; Cuba; 1969). TT^e positton taken 
here is conciliatory. 'In particular, the values of theory and the logic 
of r'i^Qrpus 'scientific research should n^ be forgot ten /ignored, or 
disdained by the adriiinistrative researdherV.'lCurt Lewin', proposition 
that "there is nothing so practical as a good theory" st|ll holds true ^ 
for the administrative researcher, as -does the des^irabi/ity of unam- 
biguous results. If .Program A is working and Program / ij^^not working. ^ ' 
it is administratively necessary to be aware of that iact. Even more. / 
helpful in tei^s of future^ policy and practice, howier, would be a ' \ V 
compelling explanation of whj, that was so. Such i/the value -of* theor^. - 
Even better^would Ije such an. explanation backed i with unequivocal ' / 
empirical support. Such is the value of theory/nd scientific research 
working together. 
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_ °f the term "administrative research" used here, 

Stufflebeam, 'etj^ (1971) use "evaluation,/ which they define as "the ' ' > ^ 
/ process of delineating, obtaining,' and pro/iding useful information for 
Judging^ decision alternatives" (XXV). The authors make a succinct 
. : statement on the underlying reasons for/similarities or differences, . ^ [ 

between research and evaluation, and j/eir statement serves as a 
^ fittfhg conclusion to this section: . . T 

■ The^parpose of research is to ,proyic/e new knowledge r 

^ . and Its mettiodology is designed to produce- knowledge that L 
• ^ ^= universally valid. The purpose of evaluation, how- 

ever, is to delineate. Obtain, and provide informatidA 

tq,r making eauGational/decisions . This information Js f 
not necessarily new knfowle^ge, and ev^n more important. 
It IS highly parpitul^ristic and specific to a decision 

situation, rather thin generalizable to many or all ^ ' • f 

, ^ f"'"^'' ' Wluation methodology is not necessarily ' 

\ designed to produc^^ universally valiil information, but 

- ' ' i"f°.™ation that v^lid and usefuf within the decision-' r 

-MlSHlg context. ^Insofar as the decision-makiT;! 'context^ ■ ■ % V 

IS highly generalizable and the intent is to provide 

^ ■ i . ' • 
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new information wlthoutj precedent, the purposes 
and methpdologies of re|sjearch and evaluation '^mav 
be equated (p. 140; emphases~in the original). 
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Types of Administrativ e Research ^' ""^ , 

\ ^ ■ -t- ' • , t 

With the generic categories of basic research and a'dministra- 
tive research introduced and contrasted ^^^^^j;^^^^ exposition continues 
now to subdivisions of administrative xesea^lrch.^ For a variety of 
situations, consider what decision it is that must be made, who must 
make it, what information or data they need to make it, and what general 
principles or rules of decision-making they^can, do, 63:i^hould. 'apply in^; 
reaching a decision. Onfe can reason through such a set of questions 
and -arrive at his own set of categories for administrative research. 
Chances are that the set would look' something like\th3 four categories 
below. Mare or fewer subdiv^siorw-eetrW^e made (ripcall' the f pening. ^ 
anecdote about tl\e three baseball umpires) but the following .c^tegori'fes 
allow distinctions to be made that are believed to be 'important: 

■ r 

A. Background research - " : 

^ B. Formative research - 

C . Summai i ve re s e a rch , a 
p. Policy research. 



Background Research i 

Something is needeS to distinguish an importantx category of 
administrative research that is or can be independent of a particularl 
TV program, product, or mission. ' It -is called "background research"-; 
here; it co^^ also be called "planning* resfear*ch" or "contextual 
research." For ETV, an important objective of background research 
would be general knowledge about the real. and potential audience. * 



ERIC 



12 



Consider otich. quo ot ions ao the follo^in^r ^ ^ . ■ 

1. What iG the coverage area '^if oiir usable TV signa*^? 

. 2. How ©any working receivers are contained within' the ^ 

signal area? Where ajre they' loqat^d? Who has access 
\ " ... , ' ^ ' . 

to them? 

3. What is the demographic compoS'ition of the totajl ' 
^ ' • \ ^ \ ' 

population, the potential TV audience, and the « ' , 

typical TV au^ence? ^ / ^ ■ 

The three questt<)ns above typify a host, of considerations that are' • • 
indispensable inlaying' out general ETV progra^s^ing strategies ration- 
ally. '• . • ' » , ' 

In' highly Industrialized societies, d^^wealth.of general 
r^feren^e data is supplied through business and government agencies. 
There are also specialized 4ata^„services for b'roadcdsters and adver- 
tlsera.^ Even so,^ special background research studies must frequently 
H cpmmissigneds With a deireas^MWcess to standard reference data 
there will be,ari increase in the need to conduct original background^ 
reseaiirh.^^ .TKro«gh data on suci questions-^s the ^^ollowing, • background 
resetarch can be halpful in suggesting programming ^eeas; ■ ^' 
Within varipu3 demographic sub-categories"^ . ... 



' 4..^^. . . .^o t?hat interpersortal qhd mass media sources 
do people turn for advice on topics>ji,'^% gnd C? ' 

5. ... . . ^h^t are the areas of greatest concern? * 

6. . . . . what' do people perceive as their greatest\informa- 

tion needs? 

.- ■ • * . 

♦ 

utilizing such questions, Mendelsohn (1968, 1969, 1971) -has pioneered 
in using. both demographic and psychological data as feedbak to ^e ) 
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planning process in the area of social amelioration progrqrajuing fo'f' 

television. In his Denver, project (1968), for example, he detemined 

through audience surveys that many mothers had sole responsibility fi^t- 

their households', were worried about money and health problems,- and » . 

didn't Imow how or x^here to fin<} assistance. These results provljded 

guideliites to the writers and producers. Mend'elsohn spmetlmes found , 

that the facts gatUered throu^j/^ such empiri<cal background research'* 

contradicted the estimates of "expert" consultants. 

BackgroL^nd research is also needed after a general program- ^ 

mtng strategy' has been determined. Suppose, for x^hatever , reasons, that 

the administrator is committed to produce lin ixv series on agrarian 

^ » 

reform, but no ^specifics as to scope or style of presentation have yet . 

<< ■ - ■ 

been -decided . Consider the administrative utility of ansx^ers to t^e 
folloxd.ng questions, for esiample: \ * 

\ Within various dempgraphrc sub-categories ... 

7, , . . . what' is presently IcnoxTO about modem agrarian 
practices? What genera-1 misconceptions are apparent? What 
are fche largest gaps of knowledge? 

8 what attitudes are heW toward agrarian reform? 

w * . ■ ■ iS 

The ETV programmer must reach the target audience member on his (the 
target audience membet's^ terms, and objective background research data 
can illuminate" the dimensions ^of this task. Consider, for example, the 
programming implications for almost any social amelioration ETV series; 
giyen a situation such as ythat found by tl^ie » Paraguayan researcher Juan 
Dizg Bodenave (1965). Bodenave interviewed 221 peasant farmers who 
lived about two hours* distance from Recife, Brazil. At the time of the 
study . (1963), Recife, a state capitol, had almost^ million inhabitants. 



Hio otudy revealed that • • . ^ 

• . • 94% of the peasants did not know that coffee 

was the principal export product bf Brazil; 

• . . 8(Ki no definition for the word "democracy"; 
^ ' • • • •"'73% had not heard of Fidel Cqstro; 

. • . . 65% had no idea of what a dollar was; 

• . . 48% had not heard the name of the President 

of Brazil, 

• ■*t 
Formative Research \. 

The terms "fbxmative evaluatiorr" and "suramative evaluation^'' 
originated with Michael Scriven (1967), although the underlying concepts 
predate these particular labels whicji are now in common use. Formative 
research/evaluation deals in general , with product ob program improvement 
it functions to provide 'evaluative feedback to the production staff 
while there is still tijne to incorporate this information and change 
the program. Formative research .applfes to the fonoation or form^ve 
stages of a program or product. Sumpiative^ research, on the other^^^d, 
reports only .after the program (or subdivision thereof) is completed;' 
it evaluated, vafter the fact, the extent * to which the pr/gram (or siib- 
division thereof) fulfilled its objectives. Stake (1967)^^dfid prer 
cisipn to Scriven's terms- by contrAsting the criteria to which forma- 
tive and siimmative evaluations are oriented. Formative evaluation is 
oriented to* "developer-author-publisheir' criteria and standards." " (In 
the ETV setting, we would probably substitute writer, producer, direc- 
tor.) Summative evaluation, ori the other hand, is oriented to 
"consumer-administrator-teacher criteria and standards." (In ETV we 
would probably add "sponsoring agency.") 

Formative research,' like all administrative research, shoi^ld . 
be an aid irl^ 4eci^ion-making, The various categories of administrative 
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research being discussed reflect various categories of decisions tliat 
admlnist^ors must make. In the ETV formative research category, the 
decisions deal mostly with message design. The major consumer of 
formative research is the production staff. The needs of production 
dictate the agenda for formative research. ^The criterion by which 
fomative research is judged is the extent to which it was useful to 
the production department in improving the program. Producers will not 
set out purposefully to make programs that are boring or incomprehensi- 
ble, but, for a variety of reasons, such misfortunes do otcur. Part of 
the problem may be a reliance on person A's opinion of what person B 
will enjoy, understand, payattention to, learn from, or be persuaded 
by; in other wor<ig; relianc>e on intuition^r advice as a substitute for 
actual target audience 'reapbipn. In evaluative ^search, there is 
ultima^rely no substitute for the reactions of the target audience it** 
'^elf. In program improvement (as opposed to performing an "autopsyV \ 
oftet t)[ie resources have already been spent), thfere fs also nO substi- 
tute for getting' this reactfion in time to IfakL correctiye-action; 
Rapid, sustained feedback to the producers, baseji on representative 
audience reaction, is probably the major contrib'utipn tliat formative 
research can make. to the administrative system. , / 

If the purpose of the* production staff vaorking with formative 
researclj^^data is to improve the program, and if the major program 

^ • ' " A. 

Elements under the control of the production ataff are th.e stimulus 
elements (i.e.,, the style and content of the TV program), it follows 
that formative research in ETV should have s>t imulus implicat^^ons . 
Perhaps it seems strange to make a special point of being relevant to 
prod^Gtiop needs, but, hlstor^ically ^ ETV research has not been very • 
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sensitive to iit he producer's ^eeds,^and producers have,, for the most 
part, ignore^ research findings. However, experience has shox^ (see, 
.for axample, Palmer' and Copnell,' 1970) that production staffs will use 
research data when it is relevant tcr their day-to-day problems and 
decisions, whicl\ is precisely the function of fo|raative research. No 
one who .has given serious thought, to the communication process would 
argue that stimulus eontrol equates wi^Iv^contrdl over effects, because 
this would ignore x^hat the- audience ^contributes to commjunication effects 
X Tlje stimulus (the program), however, is the one thing that cah be con- 
- trolled 'dj^rectly .by the ETV staf B^v/ and consideralfle attention should 
thejrefore be given to it. ' ^ 

This production orientation often leads^ to a research product 
thatt is quite different from the btfsic research described earlier. For 
example, in order to get research that can be related to the decisions 
actually made by TV producers and directors, moment -by-moment effects 
data are frequently gathered. VJhereas mo^t people would probably not 
instinctively think^of evaluating the 17th minute of a progr^ara in com- 
parison to the 16th minute, this relates precisely td" the .complex series 
.. . • 

of decisions made by a TV director. The physical television program 
ultimately results from a complex, moment-by-moment series, of production 
decisions made bjj' a^TV director. ^ To ctfodify and improve a television " ^ 
program is to change some of these decisions. In this context, moment- 
by-mom6nt formative research data function quite logically to help 
locate the specific production decisions that need to be changed. 

Consider now the issue of what effect (s) , when related to 
stimulus variables, v;ill be most helpful to the TV producer* What kinds 
of measures will yield the. most insight? Perha-ps the most obvious 



criterion measure^^ (e£f ects/measure$) would be the terminal program 



objectives. task ^ here/ would be to relate stimulus variables ^o 

performance A^riables der/ved from the goals for the program^ Cropper 
et (196^) conducted /9 study (one of a series of excellent studies), 
designed tro measure th^ amount of improvement that could be brought to~ 
Q televised lesson by/pretesting and subsequently modifying the lesson. 
Two test lessons wer^ recorded and pretested among a representative 
group/ of students. .The tests wiere based directly on the instructional 
objc^ctives. Test ybesults were analyzed item by item alongsideJplaybacks. 
of the recorded i/istruction in an attempt to discover the preciLe points 
of inotructionali inadequacy and th$ und^lying reasons for the |tudent 
difficulties, .^is analysis coj^^tituted thq recommendation for la prp- 

a ll ' \ ' ^ . . 1 

gram revision, |and a new recordfijig was made. With a considerably degree 
of rigor and contrcM^, performance' on* the original instruction was« com- 
pared with performai^e^n the revised instruction,^ demonstrating i^mprove^ 
ments in p^erformance rangingv from/ 12 to 2$ 'percent as a function of ^he 



one revision. 



Tlie situation' above Involved in-schooL^elecasts with rela- 

o ^_ / ^ . _ .\ . ^ . . ■ 

tively homogeneous groups of Vtudent^ in a supervised setting; the 

/ . •, . • - 

students were a "capti^^' audience. Th'^ heterpgeneous , anonjonous, non- 
cap ti'Ve ^audience that must be attracted to view presents yet another 
fotmatjtve research problem^. Here,^^the formative researcher may find it 
nec'fessary to work back from the terminal objectives to thV n-ecessary^ 
prerequisites of the terminal object;Lves — prerequisites suc1;^as ^ 

attention and comprehension, fnlike face-to-face' communication, there 

. . ^ ' I , • ■ 

is no opportunity in television! (except in special two-way systems) to 



detect evidence of audience ina 



btentibn or lack of comprehension duriijig 

18 
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the presentation, and thus no opportunity to take immediate cortective 
action. Therefore, attention ahd comprehension cannot simply be 
assumed. The producers' need an advance estimate of these f actors, i 
based on program pretests among Representative audience members. 

Unless the program is attended to, there is no point in 

\ 

further exploration of comprehension or achieving program objectives. 
Children's %levision Workshop (CIVI) employs a technique called the 
Distraqtor Method to determine the "attention profile" of programs or 
program segments* (Reeves, 1970). A child in such. a formative tesearch 
test would have within his field of -vision two things to look at: (1)' 
rhe TV program being tested, or (2) a'series of color slides projec.ted' 
on a rear screen, with the slides changing every few seconds. Actual 
looking behavior (toward the test program or toward the "distracting" 
slides) is measured on a quantitative scaj«e repeatedly over time, thus 
yielding the attention profile. Other recent reports that incorporate 
observational techniques of actual viewing behavior 'are Bertram (1971) 
and Ward (1971). • ^ 

One of the earliest systematic approaches to diagnostic, 
moment-by-moment' analyses of program effects utilized a device called 
a program analyzer, through which test audience members could, during 
the actual-monitoring of a program, .register interest or disinterest 
(or other reactions) on a continuous basis by means of push buttons. 
Lazarsfeld, who along with Frank Stanton developed program analyzer 
procedures, feels t.hat the procedures are still very useful for what 
in this chapter would be called formative research: 

j Perhaps suclj a device is most useful for finding 

/ out reactions to programs not yet on the air. With the 

/ proper research design- it would provide one way of 
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studying the detailed reactions of unsophisticated 
viewers to sophisticated programming, and can pro- 
\ - vide clues to the serious problem of raising the 

level of sophistication. Program a««4lysis has been 
used primarily to change the content'-of programs, to 
edit or delete ^difficult parts, to avo^d^the so-called ' 
boomerang effect?; however, it could as well play a 
role in helping to locate people's viewing difficult ? 
ties. If we could learn something about the stumbling ^ 
blockd^ less-educated people have when viewing; let us 
say, a serious play — - and the program analyzer pro- 
vides a useful technique for getting at this -- we 
could then experiment to find out ^^hether supplemen- 
tary' aids, like program materials, or an introductory 
discussion of the play, have any noticeable effect 
(Lazarsfeld, 1971, pp. 197-198). 

Belson (1967) notes the significant gap in education and 
technical sophistication that typically separates the broadcaster and ^ 
his audience. Thi^^^tir&ans 'that the broadcaster's estimate of what the 
audience can comprehend should be, if not suspect,' at least subject to 

empirical test. As is the^ case for all forroative resea^ch^ ^such tests 

"v. 

do not necessarily involve complex techniques. Belson, for exatnple, 
finds great utility in presenting program material (radio or TV) td 
people brought together specifically for the testing e^^ercise, then ' 
asking them specific (written) qvife^stions ^over the material. Co;rrect 
answers to the questions would indicatexcomprehension of that part of 
£he program. If the proportion of compreheji^oii is low, the major 
tormative research effort would be to analyze the Xpatterns of miscom- 
prehension so as to get insight intci^hat is needed to ihvprove program 
comprehensibility. -Formative r^earchers' at Child4:en's Television 
Workshop can ob^ain comprehension frqto individual subject\ori^ 

a taoment-by-moment basis by stopping the (recorded) presentatidn in 
^^^"Pjogram and asking questions targeted to the immedi'ately ^preceding 
information (Palmer, 1972). 
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J.„3tlv. r..e=«h on comprehension is not restricted to • . 
. ,er.al content; it can also probe the comprehension o. production con- >, 
■ Mentions and , variet, of nonverharcontent, Oho and Schxa- (1.67. , 
, „. ,3,.U0) discuss one. aspect of this area of comprehension: the 

p„hlem Of co^unicetins .With veihall, and/or visoaUyilUI^-^, ^ 
people, .he. Cite a variet. Of studies in -hich Pict«ial conventions - 

.ere not understood (e.S- use_ of close-ups. shp„ing only part of an- 
■ ■ Object, etc.) and conclude: '..e would he taUng too ^ch for granted, 

assumed that people having different experiences with visual ^ 
■ .mases would a film or still picture the way we do" (PP- 138-139). 

soohisticated the encoding of meaning hy way.of symbolism and 
The more sopnisticaceu i." ^ ^ 

the greater the risk of miscomprehension or laclc 
production techniques, the greater i: 

J ThonrPtlcal issues dealirvg with 
of comprehension at the decoding end. Theoretical 

..e "language., or the "gran^ar" of fil^ and television cannot he dis 
^ eussed here, hut the in^rested reader ma. refer to Wonena^S.)^ 

- " . a959)- Whitaker (1970); and Pry luck (1968) for 

(1960); Spottiswoode (195y;, wnicdn. 

■ various po^pts of view.' The point to he made here is that ^ meaning. 

intentional or unintentional, verb.1 or nonverbal, that is encoded in a ^ 
,. .elevision program should be a potential candidate for formative .search 

in comprehension. , ^ 

r.f uavs simultaneously because 
TV can communicate in a number of ways sim 

^ " ■ H 1 channels are so-unre^aine-d with this medium. Much non- 

the nonverbal channels are a 

■c ■ f^Vmal- cultural norms that are not 
verbal behavior is the product of mfoW cultura 

• K t .re^evertheless faithfully taught, learned. a,?d 
codified formally but are nevercnei 

- ' / « a Hall 1969). For the s^me 

adhered to within a culture (see, e.g.. Hall. 1 

= ;-r,' r.ilture A to insult or offend 
reasons tha. it i. possible for someone in culture 

• / It B (i e violate his uncodified cultural" norms) 
a person in culture B (.i-e*. 

er|c; - 21 - 
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througlj the unwitting exercise of nonverbal communication, it is pos- 
sible' for unintended messages to be conveyed nonverbally via TV, The 
greater the cultural diversity between producer and- audience , the 
greater the potential problem.' Again, formative research can be help- 
ful. Programs can be pretested on the target audience for the purpose 
of finding out what unintended and undesired me*ssages might be con- 
tained. Random samples -^f the target audience would usually not be 
required here, and not necessarily even large samples. . If*a norm were 
truly universal (adhered to wit*hout exce^t^n) within a cultural cate^ 
gory, it follows that even one representative audience member could 
indicate if a pretest TV program was offensive or subjectjto misinter- 
pretation. If management has more of a need for general and sustained 
feedback reactions within a cultural grbup or subgroup than for the 
specific pretest data mentioned above, it should be fairly simple tOjp— 
set up voliKitary viewing panels from the target audience whor could pro^ 
vide evaluative feedback on a sustained basis, including reactions to 
nonverbal aspects of the programming. . 

From the discussion and example's above, xt should be^ appai;:ent 
that fomative -research in ETV caatake .ma^ fojrtns. The unifjrj.ng theme 
is that such research or evaluative data should be of utility to decision 
makers in the prpduction-prograitiming area. Formative research should 
help them improve their product, the TV j^.rogram, Th^ logic of each sit- 
uation will dictate v;hether such formative^ research feedback must come^ 
before a program -is aired or whether it can still serve a useful forma- ^ 

tive research function after being broadcast (Ss, for example, in the 

♦ 

case where evaluative feedback from an early program in a series, can 
affect production decisions for later programs in the series.). 
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> ?oth background research and -formative 'research should wbrk 
toward giving the ETV program a reasonable chance of achieving its 
objectives. Research reports in these two categories are usually not ' 
Of interest to the general public or ev.n to agencies outside the 
^road^st operation, .or do they specify what, if anything, was gained 
fro. the ETV expenditure^. There may be temptations, therefore, ^to 
Place research mon^y in more "visible., categories. AH too frequently, 
critical pre-broadcast decisions are short-changed, in terms of resources 
devoted to research. 

Sunmiative Research * - 

. Summative research, as indicated in the previous section, 
assesses the extent to which program objectives were achieved. J ,, 
sponsor, the administrative staff, the consumer, and frequently the 
general public are interested in ^hese evaluations." ..fiow well- did it 
do;., is an inherently in^ieresting question. Issues of high importance 
in formative research (e.g., does format X or format Y hold attention 
better for audience group Z?) may not even be mentioned in summative 
research, because they b.Iong to a" different chain of decisions and 
decision makers. However, when.the station manager asks ..should I 

J' . 

cancel this program or con^ue it?..- when thP h^,,^ f 

, wnen the head of a government 

agency asks ..shcJuld I fund this xlT^e^rfes^or anoM,. 

IV serie».«4;Q^^^^another season or not?..; 

when a school principal ,sks ..should I conunit W^^l to this W 

series for next year or continue as before?..; the need for an overall 

assessment of effectiveness is obvious. 

' .In s6ae form, summative research must be based on a specifica- 
tion of objectives for the proeran or qpWoc n i i 

1^^°^^^^ or- series. Belson.s comments below 
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vrese j^e in the context of comprehension tejts, but they apply with 

equal relevance to summit iveL, research: 

It is necessarv^ight at the start to develop a 
clear idea abdut^mat the progTram material is supposed 
to be telling p^ple. It is not unfair to say that at 
times this can be difficult to ascertain even with the 
. script available for careful study. With lengthy or \ 
complex material it may be necessary for the question 
designer to check with the writer himself in qrder to 
ensure that his* interpretation of the message is what 
the brwdcaster intended XBelson, 1967, p. 126). 

The situation implied in B6lson's ooimnents is not'^ideal, but it may well' 
be typical. The linfortunate implication is that the essential program- 
mitig and production decisions- were made before objectives were specified 
or at least before 'the evaluation team was consulted. The situation' 
where the evaluator must- look at a finished progra^i and the^infer what 
t;^e specific objectives must have been, so he can then design an 
instrument to check performance, has happened all too often; 

A better model would be the summative regjedrch procedures 
followed at Children's Television Workshop for the program Sesame Street 
(Ball and Bogatz, 1970; Bogatz and Ball, 1971). At the beginning, con- 
tent and production experts were consulted to develop program' objectives 
that met the social need and were appropriate for the television medium. 
A few illustrative behavioral objectives for the 3-5 year target audience' 
of Sesame Street were? . . ; 

1. Matching of letters: "Given a printed letter 
the child can select the identical letter from a set 

of printed letters" (Bogatz and Ball, 1971, Appetfdix B). 

2, 'Recognition of numbers between 1 and 20: 
"Given the verbal label for a numeral the child can 
select the appropriate numeral fi:Qm_^_, set of prints 
numerals" (Bogatz and. %j:Iv B). ' 
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3# Visual discrimination; ''Given a form the child 
can find i^s counterpart embedded in a "picture or draw- ^ 
■ Ing" (Bogatz and Ball, 1971, Appendix B). • ^ 

Prom the beginning, tlie summatiVe research team w^s tsrorking on the 

test battery which was, in effect, the operational definition of pro- 

'* 

' gram objectives. The test battery would include, for example, the. 

^- . 

actual letters for matching, the numbers for recognizing, and =the 
embedded figures for 'finding. Through a. lengthy prebroadcast period 

-of planning (i^iich is ideal, but not typical) ,^ the summative research^ 

j - ■ * . '>.^- 

/ • • . f ■ - • . 

te^t battery was developed and administered, to target audience groups 
for baseline data. Summativ^e research was not an afterthought, but 
was- an integral parf of the process right from the start. There are 
many examples of'summative research -on television programs, of course, 
but the wide dis'tribut ion of Sesame Street around th6 world makes it a^ 
good soured of examples. a , 

Certain differences of emphasis on how to relate to behavioral 
objectives should be noted, especially as these involve suinmative re- 
search. For- a useful introduction to the case in favor of ^behavioral, 
objectives, see Mager (1962). The general premise on which behavioral 
objectives are supported is that one must know what he means by success 
and, through explicit 'criteria, be able to recognize .success or failure, 
when he sees it before success or failure of a program cati be determined. 
In a scientific sense, such recognition implies, observable behavior and 
accfeptable measurem^t procedures. While on its face' this seems en- 
tirely non- controversial, several problems can emerge. If there is no 
"acceptable measuring instrument for effect X, some might argue that 
^ effect X cannot be considered o>ie way or the other. Yet' such restric- 
tions can be contrary to common explirience or common sense. For 

o • • - ' \ 25 ' • . . - 
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example, it i's doubtful that a music instructor would give up hi/ I 
commitment to music appreciation as. an instructional objective even if 
this quality totally eludes acceptable measurement procedures. If • 
objectives A, B, and C are specified and agre'ed to in advance,, it is 
conceivable th^t the rigorous devotee of behavior'al. objectives would " 
ignore phenomenon Da? being irrelevant,' when in fact it might be quite 
relevant. For example, suppose the subject jiatter is chemistry, tfiere 
are three objectives,; pd A, B, and C, wliich involve co"nduc ting and 
explaining 'three expetimeVits, Suppose phenomenon D is the student's ' 
Interest in doing furJher study in" chmistry. It is 'possilile to succeed 
^ in the three experime|ts, Uut in the process squelch; any ftirthe.^^; interest 
in chemistry. OnX common- sense <basis, most of us would not label tills 
"success," xv^it^ ot the fact ^that th^ pre-specifled objectives were 
met entskl^.. There is no. single ••correct'^ position tq.take here, but 
the ETV researcher should be sensitive to the strengths and vealZsses ' 
6f whatever position he does ultimately take. See Stufflebeamj 
(1971) for a ipofe detailed discussion. If\ne understands the concept. ' • 
of process (as presented, e:g.,.in B^rlo, 1960), he -«ili: understand that 
• any^ ETV program will probably haye multiple effects , some 'of- w^lch ^' 

can be caught in any particular mea|urement effoft, and only 'spm^^ci;** 
. which will have been plannecf for as progrim objectives, andvonly s.om 
;of which' will relate dire"ctly to* the" program sponsor's area re«ponsi^ 
bility. It follows ,that sunimative research cannot be exhaustive but i£ 
Should attempt ta^rovide evidence in the, categories tha.e...axe,a^ should 
be most rel^vaiil^to the administrative decision required; . - 

X"^^ ^3ct that ideal. experimental conditions are rarely avai'la- 
ble to the a^ministrat^ive researcher/evaluator .working w{th ongoing 



^r:i:^'^s -^6^;y.t M^ "af^E t^.J^ook"; he ^st still be^ountabl. 
i^r the tnterpretab^lity of his data; he must sUU Swer'quesUbtls--. 

such aB the following: , 

1. ' Whdt assurance db we have that effect Y was- due to X 
• ' and not due to something ^Ise? ^^y sibuld we «i.nter- . 

■ pret your data' the t^ay you do?/ - 

2. ' What assurance do we have that the eVidente is consis- 
• .tent and stable? Why should' we accept your measures 

as reliable and valid? , " 

3. On what, basis can you generalize beyond the specific .\ 
units you measured? 

ExperimentU logic or sose form of an aU^nativ . i P . proxlmatlon of V. 

axperimental logic U needed to handle such questions. % 

■ Given the need for rlgor,,=uch as Wenfific experimental . 

design provides, and given the near-lmposslhlUty (or at least great, . 
difficulty) of employing true experimental designs in EIV summative 
research, whet are the altematlvesJ A workable general •principle would 
■ be to.understand the. logic of the t^e scientific experiment and -then to 
make as close an approximation to that degree' of control as, data utilit,. 
the test situation, and research- r.souices will, allow. Such a.general 
policy .should help guide the EOT researcher to generate optimal realis- 
. tic designs in a number of different settings. «hile ihere is a great 
' ™„untof literature on experimental designs, there is considerably 
less oL the' approximations- so frequently required In EOT su^ative 
. research, in the search for alternatives, the reader should find the 
follo.«ng references helpful, Belson (i^X^'; ^I'l-" (l'^'>-- 
(1969); Campbell and Stanley. (1963) ; Web^ Jt^ai,. (196654, 
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• Cross Media Studies as Summative Research/Evaluation 
/ ' the fifties and^'sixties there emerged a category of ETV 

/ • .^y* .^research, insjiructlon was compared 1^ ''conventibnal" 

instruction, Xhe,se are sometimes called *the "TV vs . .face- tcT- face" ' ^ 

' Studies^ and t;hey number • several ^undred. Summary references to these 

. 8tudi,6s can be found in Lumsdaine and May (1965); Reid anA MacLennan 

(1967); and Chu a^Schr'amm* (1967),' In terms of the research categories 
being used in this chapter (basic research, ' plus four categories of 
administrative research), many of the "TV vs. face-to- face" studies 
would have to be classified as summative research. ;TV vs. face-to- 
face" studies ha«\^e been repeatedly and severely criticized on both 
methodological and conceptual grounds (see, e.g., MacLean, 1962; 
Stickell, 1963; Kittross, 1967; Mielke, 1968,. 1971), and indeed, such 
cross-media comparisons are now rare. 

To Judge the adequacy of a Methodology is to check on sudh * 
features as the research design employed, the degree to w^ich assurap- . ' 
tions of statistical tests were met, and the like. Although admittedly 
difficu^lt to apply in on-going processes such as schoolroom classes, 
hi^h rigor is not difficult tp recognize," in the sense that it is de- 

■ fined by rather widely accepted criteria. On methodological grounds 
alone, Stickell (1963) found only 10 of g50 comparisons of "TV vs." face- ' 
-to-face" to be "interpretable." » * , • . \ 

It is somexi/hat m^re difficult to judge the adequacy of the 
question asked in the" first place. Basic research seeks to tesf'a 
rationale or theory. Good basic research questions are formed so as to 
conceptualize clearly a reason for expecting one variable (or set of 
variables) to have a demonstrable influence or effect on aflother / 

■ ' '., ' ■ ■ ■ 28 ■, / ' .'^ , • 
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variable (or set of variables) • The "TV vs. facib^to-f ace" ^studies have 
^een impreJsively lacking in such rationales, and they have , made little - 
If any contribution to basig theory. 

4 v.,. ii^ ,E2g)loratory^ oregfearch ^ vjhich is pre-thepretic, seeks to dia- 
cover pta'ble relationships among variables so that raTional&s'' far s these ^ 
relationships can subsequently be derived and tested. Good exploratory 
research questions are therefore questions that have a high or reason- ^ 
able probability of payoff, i.e., of finding stable relationships that 
can be explained^ with a scientific theory. For example, consider the 
scientific study of weather. For exploratory research ori the weather 
to lead to general theories and principles o£ weather phenomena,- at 
least two things must happen: (1) theoretically relevant factors about* 
the weather must be observed in the first place; and (2) these observa- 
tions*must be made systematically and with enough precision to allow 
patterns and relationships to become noticeable. The same reasoning 
holds for ETV- related phenomena. Both lack of precision and lack of 
theoretJLc potential have plagued the **TV vs. face-^to-f ace" studies, and 
little in the vJay of new or modified theory has emerged as their result. 

Administrative * research serves administrative requirements 
rather than 'theoretic requirements (although these two requirements can 
sometimes be found to be closely associated). The good administrative 
research question is one th^t serves well the needs. of the administrative 
decision maker. Even here, the utility of the "TV vs. face-to-face" 
studies is lacking. Tii^y serve no obvious background, formative, or 
policy research function,^ and their summative research function is 
severely limited by some of the specific problems discussed below. 
These historical problems with "TV' vs. f^ice-to-f ace" studies illustrate 
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pltfallo in rationale and procajdure, that can be repeated in nex^ settings * 
6f (lusnaativQ research in ETV. 

• A, "l£ TV 'a£ good as ' face- to- face?" i£ £ poorly conceptual -' 
ised question , possible reasons for expecting TV to bo^-^'^tter" (e.g. 
fci^TC released time for lesson preparation;\abilii:y to "show directly" 
rathV£j:hpn^''talk^bout''; distributing the best ins^truction. to all ' 
students; etc.) or possible tjeosorfs for expecting TV^tV be *^/5rse'* - ^ 
(e.g., inability of students to question the television teacher directly; 
inability to adjust lesson pacing to the needs: of individual students; 
etc.) might be useful factors to study, because they deal directly tjith 
characteristics, correlates, or consequences of the television medium 
or the television system . To ask the unrefined- comparative question, ^ 
however, is to ask whether unknovjn package A is better or worse than 



unknovm package B;^ho matter x^hat the. "ansx^er" is, it will be very 
difficult to understand, ^explain, utilize, or generalise. 

B. The more the two treatment conditions are matched , the 
less realistic either condition . *The more the TV and face-to-face 
groups are actually matched on all factors except the type of mediation, 
the less freedom there isyfor either form of instruction to do what it 
can do best. Conversely, the more each form or system of instruction 
escploits its unique potential, the more difficult or impossible the two . 
become to compare and analyze scientifically. Conceptually, the problem 
of scientific assessment^ becomes amplified if TV is but one component of 
an innovative system , as is' tlt^^ase, for example, in Great Britain's 
Open" University (Smith, 1972). Pot^tially, this introduces a^ double- 
edged problem of (^) not being able to detect an effect unless the 
entite system is activated; and (B7^J:^n not being able to attribute 

' ' "• ^' 30';' ' ' ■ - 
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effects to specific components in the system, |uch''a8 television. The 
final report of the National Project for the Improvement of Televised 

Instruction, 1965^1968, notes: 

,. . . in any compli: situation if only °ne variable 
is changed no significant difference results. This 
' is why the'gverall improvement > American education. 
aft"r ten to fifteen. years of "spQt" innovations has 
bfen relatively small ^Tn..rd . Significant Difference, 
."i ' "-undated rApoyt, P; ,36).^ «^ / ", ' 

The .other- side of the dilemma is articulated*; by Vttentag (197;) who . . 

writes: ^ 
■In contrast to an experimental study, a' researcher 
ev&luates the effect a programme.. . . not pne 
effect of a variable. Prograsiimes . however, consist 
of a multitude of bits «and pieces. ^They are the 
inverse of the carefully defined and manipulated 
singU variables of the experimental paradigm 

(P- 76). / ■ - ' 

C. Even when each method is free to exEloit what, it does best. , 

the selection of the " effects " to be measured (the. de^end^ variables) 

•f^eauentlz cannot re^ 
this problem relates to the -1>roblems of specifying behavioral objectives, 
discussed earlier. Just because an effect cannot currently be specified 
and measured does not mean that it is irrelevant. Just because an effect 
v,as not measured does not mean that the effect did not exiat. On meas- 
ures of cognitive lining, one will typically find no statistically 
significant differenceVD) between the TV and face-to-face groups.' 
Other studies also fttidAwith home study groups .nd'pther f<,tms of 
instruction. One could ^^^argue that (a) the measures of cognitive 
learning (typically paper-penXeasures of factual recall) .tap .an-^ 
impressively robust quality in a variety of settings; or (b) such meas- 
ures are simply insensitive to. ptheVdifferences that do exist . .Let the 

31 



365. 

researcher examine such an instrument designed to test c\>gnitive 
learning and ask himself if there is any gpod reasorTtb expect a TV 
group, a face-to-face group, a radio group, a correspondence study 
group, etc., to have a marked advantage* in completing such a test 

successfully. If the answer is "no,"' then that particular instrument 

'■ - ' • ■* 

' is ifltohably not well suited to detect possible differences among 

various methods pf instruction^ such as television. 

What might some of the qlusive differences be i.e., 
effeci*^r.iJ^4t would probably eludd traditional tests of cognitive 
achievement? No special case is made for tVs particular iilustratlSre ' 
list of effects, except tlie argument (a) that\hese factors would not. 
be included in a typical "learning" assessment; (b) that such effects 
could be of critical importance to the success of an ETV program; and 
(c) that thfere are situations where TV could conceivably hold a dis- 
tinct advantage on such factors. Consider these possible effects: 

1. ability to relate and apply the .information 
to one's own life (Given a dramatic TV presentation, perhaps a student 
could identify with a TV character and be greatly affected on this 
factor.) ' ' ^ 

2. credibility of the information (Some studies 
/e.g.. Roper, 1969J indicate that differing credibility le^i^els are 
attributed to various media.) 

'if ^ ' 

3. impact on attitudes and values ^(In a communica- 
tion with persuasive intent it is usually possible to "learn" the 

/ ■ ^ . 

various ajrguments presented without changing opinjLon.) 

'• ■ / ■ • ' 

4- amount of status or-prestige attributed to the 

, instructor (A frequent phenomenon is that prestige is associated with 

erIc ■ . ■ - ■ 



appearance on TV^) ^ 

j . 5. amount of incidental learning that took place 

e combined audio and video channels of television typically provide 
k wealth 'of information considered incidental or unrelated to the major 
thrust, of the lesson, at^least as defined by teachers. As anyone 'knows 
ytfiho hag spoken with yori^g children who have been "i^xposed to a 'substan- 
dial amount of entertainment TV programming, however, such incidental 
lleamlng -can be significant ,0^^ 

The list could be greatlV these few entries can 




^erye to illustrate non-obvious ef fec?Kt^ that could escape notice if th6 



only opportunity to register an effect wa\ a cognitive leai^ing mei^m^, 
Even if edch media system exploits what it does best, this may remain 
unknown. and unappreciated if the ctiterion tests\are insensitive to 

unique contributions of each system. 

< \ . ' ■ 

^» Is "TV vs. face- to- face" studies , there x^^^ an implicit 
assumption that the criterion for TV to " measure up to " is\traditional 
instruction . Such a comparison does not ask whether the traditional 
instruction is satisfactory in the first place, nor does such a compari- 
son serve any dia^ostic function for either group. 

In the more general field of educational evaluation, group 
comparisons have been similarly criticized by Cronbach (1963). How- 
ever ScfeLven (1967) still argues that when the administrator wants to 
f 

know if method A is "better" thar{ method B, the direct comparison is 
the best way to find out. Scriven argues that* demonstrated superiority 
of version A can allow a series of useful administrative decisions even 
^f it is unknown why version A was superior' and even if *A was compared 
,to B as an expedient because-^ no absolute* seal© of * evaluation for A was 
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available. For the ETV situation, the weight of eKperience In evalu- 
ative reoearch vjould seem to be against reviving the comparative study 
of TV VS. face- to- face, but this wauld^not rule out the comparative 
study of TV version A vs. TV ve^^^n B (e.g.. Cropper and Lumsdaine^ 
1961) which would more nearly satisfy the criteria of experimental 
design. 



Policy Research 



. . .if television is >chpnging its viewers, it is ' 
highly desirable that tll^- public ^nd those who 
control television should be awafe of the nature*" 
and the extent of these f changes and of pVecisely 
^how they come about, sich informatidn s^ms to 
Che writer to be essential in any attempt to con- 
trol the television medium in the^interests of • " 
society (Belson, 1967, ?. 227). 



The four, categories of EtV 'aaganistrative xfeafiaSf cttediscussed 
In ^his chapter (background research, formative, summative, arid policy 
research) correspond roughly and {imprecisely with the four categories 

of ^T^ ^research developed by Belso 

^ - \ ^ " ^ 

hensi%ility research, effects of 

research. Of the four, the fit is probably best between our category 



1 • , 

(1967): planning research, compre- 
jspecific programs, and social impact 



of "policy research" and Belson*s 



"social impact" category. If summa- 



tive research can be conceptualised as feeding into administrative 
"recycling** dedision-maUing (Stuff lebeam, et al . , 1971) ^ then policy 
research would be providing input; to recycling decisions at a more 



general societal level. 



Many policy issues invo^^xng |:elevision become policy issues 
primarily because of the magnitude of TV- related phenomena. The magni- 
tude of investment irf hardware, the magnitude of TV's reach, the magni- 
tude of the amqun^t of time devotjed to watching TV, the magnitude of the 
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opportunities i*f.TV is used creatively and xvisely these magnitudes 
raise serious policy issues that transcend the broadcasting organiza- * 

tion. Indeed, such policy issues extend to the- highest levels of > ^ 

« ' ■ ■ ■ 

government and other societal institutions. 

Sometimes, inresponse to articulated problems and needs, 
policy research is commissioned by' policy-responsible groups such as 
government. The U.S. Government, for example, has in the last few 
years received reports from such Commissions as: , 

1, The Commission on Instructional Technology (1970); 

2, The Commission on Obscenity and Pornography (1970); 

3, .TJie Surgeon General's^ Report on Television and Social 

Behavior'*(1971). ' . ' - ."a ' 

w ■ / ^ ■ • ' ■ ft- 

In varying degrees, such reports, which ter\d to be combinations ^of . 

Research reports *arfd expert t^tJLjnony, have influenced pdlic^^Jinidss/ - 

' ' ca, " ^ ; ^ 7*-.,. 

communication, although the 'level of influence is rarely as fereat a^ 

that advocated by social scientists, f « 

, Sustained monitoring/evaluation of media performance ^(see , - 
e.g., Lassvell, 1972)^ could be an example of policy researcli*. Non* * ^ • 
obvious effects of mass communication (e.g., effects on physical well- 
Jieing, eating habits, church attendance, amount of conversation, etc.) 
frequently have policy, implications much broader tl^an°the domain of any 
particular ETvl)rogram, Such policy- relevant effects would, in Lazars- 
feld's (1971) view, have long-range repercussions . A long-range 
yepercussibn means an effect that is not only sustained over time, but 
• pne tHat "also spreads out into the social field: an effect may act 
not only on one person but on several and, finally, on entire institu- 
tions as, 'fox example, on education" (p. 192)'^' " . 
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\ 

lit thq general and idealized case, the advocate for policy. 

- s ■ - 

research wotild be an advocate for basing policy on fapt, for empiric- 
ally testing many premises and assumptions of policy statements. If 
grounded in scientific method, ^such a policy research advocate would 
already have analytic experience with the concepts of multiple causa- 
tion and multiple ef^ects^ so he^would avoid oversintplif ication of 
policy issues, /The policy research advocate would be willing. to change 
pdliciets on the jbasis of evi^^dence in appraximately the s^me way 
that scientist/ would be willing to tevise theories on the basis. of ^ew 
evidence. ^ . . • / - , 

Fin^l Notes ' , \ . 

No matter how frustrating the requirements of experimental 
rigor may be to , the administrative researcher, an experimental , type 

V • ' ■ 

Of question will only yield to- an experimental type of procedure and 
logic (i^»e;, a true experiment of some , approximation to an experiment). 
If one would reje(2| the experimental logic, he would, to his impoverLsh- 
meht, alsio have feo reject the type. of questions which only experimental 
J.ogic can deal witl?i^ On' the other hand, no matter^ how ^uch the general 
rigor or generaliziability of evaluation studies may be criticized' by 
the^basic researcher, experimental methodology simply cannot cope • 
realistically with many adminis\trative questions and problems. If one « 
would f eject all* questions that ^re not rferiyed from a general theary 
and that will yield, to an experimental design, ^*^wSii;M be timing 
many administrative decisions 9ver to^ tradftion and guesswork. The 
actainistrative decisions must* be "made, with or without research data .. 
input ... \ , , 

\ ■ ■ • . ' ■ • 
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A major judgmental issue, then, , is to decide what question 
tt i?l8 that must be answered*. Once this is determined, .methodological 
choices are at least'partially delimited. Superiority or inferiority 
of a research method cannot be established as an inherent quality, but 
it can be established in terms, of performance in answering the question. 
Given this view, debate on the relative merits of "basic" vs* -"applied" 
research per se, or "foijnativ«" vs. "summative" evaluations' per se, will 
be recognized as generally futile. Real world problems will pos^ a 
variety bf questions, requiring ^a variety of research methods in re- 
sponse. 

As a means ^ ofr tyin^ together the considerably variety of 
analyses and arguments on ETV research presented in this chapter, con- 
sider a* hypothetical case of a decision to produce a TV series on £ub- 
j^ct ^. From a very large set of possible questions, a few representa- 
tive .questions have been conjectured here tt^Uustrate both the rele- 
vance and, interdependence .of various research ^puts. Brief comment 
will follow each question or set of questions. 

Q. 1-a: WHAT IS KNOWlf ABOUT SlfejECT X? 

Q. 1-b: WHY DO PEOPLE HAVP TBE BELIEFS ANd'aTTITUMS 

DO ABOUT X? • " ' ^ 

Q* 1-c: HOW RESISTANT TO CHANG5 SHOULD WE EXPECT THESE ^ 
BELIEFS AND ATTITUDE§ TO BE? 
Thes^ three illustrative questions ,r^lect 4:he need 'for a theoretic 
grasp Ot the subject matter as well as a theoretic grasp of the rela- . 
tionship of the -audience to th/ Siibject matter. Basic research;may not 
be the typical mission^ of the ETV^^^earcher ass/gned to a broadcast 
operation, tut basic researched Nscientlfic' theories founded on auch . 



. . ■ ^ ■ >37l 

research can be very Useful for practical decision-making in ETV. 
Suppose "subject X" L.s malaria, and ETV is being considered as an 
instrument to wage* a nation-wide campaign against malaria. Imagine 
the difficulty or evein futility of trying, to implement a TV campaign 
In the absence of kncrwn or assumed, answers to these three question^. 
The real issue is prz'bably whether the operating assumptions of the 
ETV programmer are or are not to be supported by scientific theory 
and basic research. 

* Q. 2: WHAr ARE THE COMMUNICATION HABITS OF THOSE MOST 
IN 2SED OF KNOWLEDGE ABOUT X? 
This kind of backgrcurad research can be very useful to the ETV pro- 
gramraer. jAt a commor. sense level, such background research helps one 
to avoid/certain pitfalls; e.g.: 

A) If the target audience cannot read, don't depend^^tt) 
"iMrJLnte^ program supplements; 

B) If the 'target audience is in bed at 10 o'clock, don't 
schedule your prograzi for 10 o'clock; 

C) . If the target audience is most fluent in dialect B, 
don't broadcas^t your program in. dialect A* 

In a positi.-ve vein, knowledge of ^bijimuiiicat^iri habits^ can be 
critical in planning general strategy, as exemplified very well in 
some of the work dotl^ by Harold Hencjelsohh (1968; 1969; 1971). If the 
vast majority of the target audience alre^y is exposed to television 
broadcasts, that Is ^rTimportatit and positive ^ctor in, favor of using 
television for the c^paign or social development program on subject 
X, if the target auLlence reliably seeks advice on subject X from a 
certain jperson or a certain institution, it implies to the TV strategy 
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planner that the help of.; this certain person or institution should be 
enlisted from the beginning. The general thesis here is that the 
target audience never exists in a vacuum; it always operates in a 
complex social system, and the objective is to work as compatibly^s 
possible with, rather than against, the audience's existing social 



system and habit p at terns j.^*-^ 



Q* 3: CAN'^'THE NATURE POTENTIAL OF THE TV MEDIUM BE 
* ' - . ■ • * 

' USED TO ANY SPECIAL ADVANTAGE FOR INFORMING THE v 

^'^'TARGET AUDIENCE ABOUT X? 

This question could be dverlooked by the planner who conceptualized 

primarily as an effici^t distribution^jysteifi,^2.^ne''Olj^ to 

^ ■» 

. the potential-ofi-^riii^ediifflv ^ an examination of production capac- 

ities^ Of the dozens of things that a creative producer can do with 

TV, which can be^sed^with greatest artistic. or pedagogical advantage? 

Slow. motion, rapid motion, time-lapse photography, instant repl^, and 

multiple images are just a few of the tools at the disposal of the TV 

(or tths^ producer/director^that are not commonly available in other 

media* Planning with sensitivity to the medium will include Conscious 

consideration of the .uniqiie me§^sage design features made possible by 

th^ medium. For an analysis of media characteristics and their impli- 

cations for message design, see Bret z (1969). ''Perhaps a less obvious. 

^iipproach to' the nature of the medium and it3 potential advantages for 

Pj0gramming about subject X would be through ^ sociological/psychologi- 

cal^ analysis of the TV reception environment. Sometimes a weakness 

from one point of view can be a strength from another. For example. 

Palmer (1969) notes that in the Sesame Street series a polf^ntial dis-\ 

advantage (being unable to provide immediate corrective feedback) was 

. • . ' ■■ . ■ ' \ 
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. at Ua« p,rtl.u/o«=e. a ccoXIar. advantage Jp.<^./, „4 
not threatening; poof perfomanca ^a, not rUicSlad or p6nlahed) " ' > 
«hen viewing u done u a.aU. private groups, a' person „ay vjlllng- 
t. receive Instruction or other co«nication on topics th-^yiTpr' ' • 
She »uH he unwilling or too eaharrassed .o receive in a .ore puhlio 
siting. For e^aeple. topics related' to persona^ hygiene, sex^duca- ■ 
tlon. and venereal dise-ase>ight'he ahle lo e:,ploit the nature of tele.' 
»l8ton rather than be handicapped by it. ' ' , , 

Q. 4-a: WHAT ARE REALISTIC OBJECTIVES FOE A^CT SERIES ON Xt 
0. 4-i:- HOW CAN IMESE BE fEANSLAIED INTO .PROOFING . 
CONCEPTS? ^^s^ 

l^e Proble. is. not only t.o-d.W™ine the Objectives 
■ which requires knowledge .f- the subject .atter and .%s relationship ' 

With the i^ended l.,^er, but. to det.r^^ne realistii objectives and , 

curriculum for televisi^ ^'^.^^^^^ the .^^.i^^elt briefly 

•:d£scussed abov. in.Q. 3: lcn6wledge oi/the nature and potential 6t the ' 

talevislon .ediu.. For pa.ple, kis r^^ement was recognised ^' " 
the.planni^g seminars when the children'-s program Sesajne Stre^^was - 
bei^jg conceptualized, as reported%^esser (1972):- 

selvfrJn ""'^ discussions we did not aUow our- ^ 

selves to „be constrained by wW the production staff 
" thought feasible in the production of a shaw \nJ 
we came to setting priorities this became^ a ^ri^^^v 

us that our task^was the concrete and pracSteal one 
. . of constructing a set of educational goals could 
■ ^^^ -^^derstood clearly and ccmv^rted into actual teJe^^ 

Vision programming. They, were^specially InsiX ' 
- ,^^;:^°'her seminar parti,5fpants uLd esoL'c S^ 
On these occasions, the skaff^^e^ley 3 Gr^^^^^ 

cnatf What dq ypu mean h3| Isjhat?" (pJ234). - 





Questions 3 gnd 4 represent types of. Questions and issues that need to 
^ be grappled with in-the admini^strative process; they do not necessarily 

require the execution of original research, although original research 

vould be entirely, appropriate. In the category scheme used in this 
. chapter, such research would probably be considered a'very early stage 

of fornfative research^ " * . 

. • \.Q. 5: OF FIVE AVAILABLE TALENTS, miCH ONE WILL THE TARQEt 
AUDIEN,CE\FIND MOST INTERESTING AND CREDIBLE? 
-This question is intended to represent a host of situations where the ' 
ETV administrator must make- a selection from a finite set of realistic 
options-. Suppose a station is trying out five newscasters for a single 
p^osition. Fomative reseatch .can provide data input to be considered 
alongside other inputs irf tive decision-malcing process (that is the 
decision on which Wscaster to hire. will undoubtedly be based on more 
factors than interest and credibility ratings). - TO the' theorist, this 
question would have ifttle- iqterest because the variable represented by 
the five options cannot" be interprejted in k theoretically meaningful 
sense.. To an experimentalist, this question wbuld also have little 
interest because the five choices do not constitute a manipulable 
•variable; the -question asked is.inot an experimental kind of question, 
Buch as, what is the effect o f ^ on Y. The fonp'ative researcher would ; 
probably agree also that the (ju^stion is not pr9found, but he would see 
that., a' decision had to be made^ ai^d that he could contribute to the 
rationality, qf thai decision by \ providing evidence. The, question seems 
to call for some form Sf surve'y ietho^gy, ^ The formative researcher 
noi/ faces V series bf sub-decifeions : how much t.ime and money can 'he 
ifevote to answering the ^\iect ion i <'and, withie -those constraints, how 
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mach tiu. ,„d aoney shjuld, he devote to answering the",uestlon? 'u 
^ :theMrin6 decision ™„st be n,.de tomorrow, In elaborate aarvey ia of 
^ courae, out ofVbe <|ueatlon. „ there la tl»e. the aurve, be 'ruled 
.out hy loo. Of b„^,et. Zf tl„e and budget, It n.a, be ruled out or co.- 
pr<»laed aa a valul^ Jud^ent on the relative Importance of this expW 
dtture veraua oJher\antlclpated expenditurea for the formative reaearch 

budget later on. Almbst bv def^n^Hr.„ -u r ' 

psc^oy definition, the formative researcher deals 

continuously rn comprpinise. • • • ' , ° 

Q. 6: BEFORe\tHIS PARTICULAR PROGRAM ON X IS RELEASED ' ' 
FOR BROADCAST. WHAT REASONS DO J HAvi TO ^OPPOSE ' 
^ "\ THAT THE TARGET AUDIENCE WILL FIND- IT. APPEALING. 

INOFFENSIVE, COMPREHENSIBLE, MEMORABLE, AND " ' 
EFFECTIVE? 

A nu-ber of ,ueatlona have been grouped Into the alngle ..ueatlon above 
/because It repreaenta a broad and important *;ea of formative reVe;rch: 
progra. pretesting and Improvement. Such' for,„atlve research la e,^. 
Pineal in the sense that it seeks to test assumption's. If ' a writer. - 
f.ela that a certaii cri-tical scene wllir as desired, be very f„„„y, . 
the fonnative researcher can try it out on a test audience and see If 
anybody laughs. Such formative research is pra^atic in the sense 
that .recommendations will be based on what has or has not worked 
successfully, as determined from all available sources, even if this 
is not^et fully-understood in a theoretical sense. Such pretesting 
can b. oier the specific terminal objectives . fir! the program or series; 
it cap be directed towar^ a seiies of preie,uisite or Instrumental ' . 
objectives such as attention and comprehension, anrf it can serve, as a 
»arnifg system for other problems totally, unanticipated, such as an" 



unintentional but offensive vioXati6« of culture noms. ^-1.: 
^ ^ ^ M» l-y-. DID THE TARGET AUDlta REACH IME DESIRED ' ^ V 

^ LEVEL OF PERFORIIANCE OR KNOIJLEDGE ABOUT X " 
BY THE END OF THE SElRIES? 

» ■ ■ s 

Q. 7-b i HMI HOUID' THE TARGET AUDIENCE HAVE KNOWM ABOUT 
■ * ?^ ME ' W SERIES -HAD NOT BEEN SHOWN!- ' 
. . to these ,ue.ti,ns are ™e„6at like the «„el score 1„ . g»e- 

■ theyandlcate whether you ,«„ or lost. . Achieving progr*. objectives' ta 
. the goal toward which. all the theorizing, background research, and .pre- 
. testing have been'.directed. The s'inplistlc. athletic analogy breaks'' 

down rapidly as actual ciplexlties are considered, such as the variety 
of objectives and the variety of>easuren,ent approaches .that, might be 
applied to a progran,. For example, the adiinistra^r -ay know that he 
• succeeded ^n crlt^ion A;, know that he failed on criterion B; be forced 

■ to rely .on expert'.oplnion on how he' did <,n criterion cV be unabu to 
afford to find out hoi he actually did on criterio'n D; and have no idea 
how he did on Criterion E. If the program Is expensive and/or contro- 
versial, there Will probab.ly be critics actively searching for nega'tive 
evidence and finding ,ault with the sup„ati.ve research report. 

. The "Four R's" for suA-ative- research might be called Rigor; 

•Relevance,, Realism and Responsibility. These values do not nece^s^rfiy 

call forth the same. kinds of activity.,, (Although no speoial questions 

were generated to' illustrate policy research. the,"Four R's" apply 

equally well to policy research.) Rigor seeks unequivocal results. '. ' 

Relevance demands that 'the answers be useful to actual decision needs. ' 

Realism recognizes that many beneficial r^fnr^e ..a - / ' 

udu/ uenerxcxai reforms and innovations depend 

ort political values as well as sriPnV,- f i « ' 

weii as scientific values. Responsibility tries 
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to find out what i£ happening and What could happen, so that a thor 
ropghly considered value system can direct the choice of what should 
and will happen. A sensitivity to all four of these values is \ 

• ■ . \ 

reflected in the following statement by Campbell (1969): \ 

What is • • • essential is that the sdcial 
scientist research advisor understand the 
political realities of the situation, and 
that he aid by helping cjreate a public demand 
for hard«headed evaluation, by contributing to 
those {Political inventions that reduce the 
. liability of honest evaluation, and by educating 
future administrators to the problems and possi« 
bilities (p. 409). 
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